Image matching device

ABSTRACT

An image matching device for extracting differences between an image taken of an actual object and model data with high accuracy. The image matching device includes a model data storage unit for storing model data of a layout of component parts of a plant, an image data storage unit for storing, for example, data of images taken of the plant, a parts library containing data on parts, heat insulating material library containing information of heat insulating materials, a model correcting unit, and a matching unit. The model correcting unit produces, based on model data and parts data, synthesized image data of an image viewed from the same position and in the same direction as actual image data is obtained by taking a photo of the plant. The matching unit matches a synthesized image to a real image.

BACKGROUND OF THE INVENTION

[0001] The present invention relates to an image matching device, andmore particularly to an image matching device for matching to an actualobject image an image synthesized based on three-dimensional datarepresentative of plant equipment.

[0002] Image of layout of plant equipment such as piping, which image isformed by data such as of coordinates of positions of parts of the plantis imaged or displayed often different from the actual object image.This occurs if model data is not modified each time some alteration ismade to the plant. In such a situation, such a model data cannot be usedwhen some trouble occurs in the plant and it is necessary to detecterrors involved in plant constructing works. Therefore, it is highlyimportant to distinguish or detect differences between model data andactual objects data.

[0003] An example of a method for collating such plant equipment datawith that of the actual plant to match data to actual state is disclosedin JP-A-9-81778. This method uses three-dimensional model data and aphotograph of an object, and modifies the three-dimensional model datain accordance with the photographed object by comparing data thereofwith transformation data, which has been obtained by transforming themodel data with respect to positions corresponding to the taken orphotographed or taken object.

[0004] The layout model data of a plant does not necessarily match orcorrespond to shape of the external appearance of the plant. Forexample, with the pipes that account for the greater part of the plantinstallation, most of the pipes are wound in practice by heat insulationmaterial, but the model data does not include data on the heatinsulation material.

[0005] Data regarding the equipment of complicated shapes, such asvalves, is stored in the form of simple codes in order to reduce theamount of data. For this reason, accurate comparison cannot be achievedby simply comparing model data with a photograph representing a real oractual state of the plant.

SUMMARY OF THE INVENTION

[0006] An object of the present invention is to provide an imagematching technique to accurately detect differences between model dataand a picture of a real or actual object.

[0007] To achieve the above object, according to an aspect of thepresent invention, the following process is carried out. This techniquefirst obtains three-dimensional model data including identificationinformation about respective parts of an aggregate formed by a pluralityof parts and information about their positions in a three-dimensionalspace, image data on an image taken of the aggregate, image informationincluding information about the position of taking the image andinformation about the direction of taking the image, and partsinformation including image data on the parts. By using thethree-dimensional model data and the parts information, an image of theaggregate is synthesized by taking a view of it from the same positionat which the image of the aggregate has been taken. The image data onthe image taken of the aggregate is collated with image data on thesynthesized image, by which it becomes possible to detect differencesbetween model data and a picture of the actual object with highaccuracy.

[0008] Other objects, features and advantages of the invention willbecome apparent from the following description of the embodiments of theinvention taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009]FIG. 1 is a diagram showing the structure of a plantthree-dimensional image matching system 1 according to an embodiment ofthe present invention.

[0010]FIG. 2 is a diagram showing an example of hardware structure torealize the plant three-dimensional image matching system 1.

[0011]FIG. 3 is a diagram showing an example of model data stored in amodel data storage unit 10.

[0012]FIG. 4 is a diagram showing an example of image data stored in animage data storage unit 20.

[0013]FIG. 5 is a diagram showing an example of parts data stored in aparts library 30.

[0014]FIG. 6 is a diagram showing an example of heat insulating materialstored in a heat insulating material library 40.

[0015]FIG. 7 is a flowchart showing steps of a model correction process.

[0016]FIG. 8 is a diagram showing the structure of a plantthree-dimensional image matching service system 8 according theembodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS

[0017]FIG. 1 shows an example of a structure of the functions of a plantthree-dimensional image matching system 1 according to an embodiment ofthe present invention. The system according to this embodiment comprisesa model data storage unit 10, an image data storage unit 20, a partslibrary 30, heat insulating material library 40, a model correction unit50, and a matching unit 60.

[0018] This system can be realized by using an information process unit,such as a computer system. More specifically, as shown in FIG. 2, thissystem can be formed by a computer system, which has a centralprocessing unit (CPU)) 203, a main memory unit 204, external memoryunits 206 and 207, an input unit 201, and a display unit 202 mutuallyconnected through a bus line 205.

[0019] In the computer system in FIG. 2, the model data storage unit 10,the image data storage unit 20, the parts library 30, and the heatinsulating material library 40, which are shown in FIG. 1, are formed inan external memory unit 206. The model correction unit 50 and thematching unit 60 shown in FIG. 1 are realized when a model correctionprogram 50 a and a collation program 60 a stored in the external memoryunit 207 are read into the main memory unit 204 under control of the CPU203 and they are executed by the CPU 203.

[0020] It is possible to provide a plant three-dimensional imagematching system by incorporating data stored in the external memory unit206 and programs stored in the external memory unit 207 into thecomputer system including the input unit 201, the display unit 202, theCPU 203 and the main memory unit 204. This system can be formed byobtaining model data 10 a, image data 20 a, parts library data 30 a,heat insulating material library data 40 a, model correction program 50a, and collation program 60 a from recording media, such as a CD-ROM, aflexible disk, and a DVD through a network and installing them into thecomputer system.

[0021] Description will now be made of various structures relating tothe three-dimensional image matching system 1.

[0022] The model data storage unit 10 contains model data, includingthree-dimensional layout data of a plant, service conditions of theparts, and type data. Model data is input by application software ofthree-dimensional process CAD, for example. Model data may be dividedinto predetermined units (piping units, or certain ranges connected bywelding or by flanges, for example), and separately stored in the modeldata storage unit 10.

[0023]FIG. 3 shows an example of model data. Description will be madereferring, for example, to a plant formed by connecting parts andequipment, such as pipes, valves, pumps, etc., and a fluid flows throughthe pipes. Model data includes part ID 11, kind of part 12, typedescription in the same part kind 13, reference points 14 (referencepoint 1, reference point 2, reference point 3), diameter 15 showing pipesizes, part model 16, service conditions 17, and insulation thickness18.

[0024] The part ID 11 denotes information to uniquely identify any oneof the parts, such as pipes and equipment. By the part ID 11, it ispossible to know relations of parts to be joined in the plant. The partID, for example, is described in a format of plant name—systemname—system number—element number. The element numbers are assignedsequentially from upstream downwards.

[0025] The kind of part 12 denotes kinds of parts. For example, PIPEdenotes straight pipes, ELBOW denotes elbows for pipe connection,REDUCER denotes reducers, VALVE denotes valves, and PUMP denotes pumps.

[0026] The reference points 14 denote the positions of each part of theplant, and include three reference points 1 to 3. Each of the referencepoints is expressed by three-dimensional coordinates (x, y, z) relativeto a predetermined origin. The reference point 1 denotes the centerpoint (start point) of the upstream side, and the reference point 2denotes the center point (end point) of the downstream side. Thereference point 3 denotes a point remotest from a straight lineconnecting the reference points 1 and 2. For example, for a curvedpipeline, the reference point 3 denotes a point where the pipelinebends.

[0027] The service or use conditions 17 denote conditions for use ofeach part, such as kind, temperature, flow rate, pressure, etc. of theinternal fluid. In the case shown in FIG. 3, the temperature (T), flowrate (v) and pressure (P) of the fluid flowing in the piping system arepredetermined. The setting of the service conditions may be omitted.

[0028] The heat insulation thickness 18 indicates the thickness of heatinsulating materials applied to pipelines and equipment. The heatinsulating materials are materials to cover the parts. The insulationthickness 18 may not be set. If the heat insulation thickness is notset, whether or not a heat insulating material is applied or thethickness of the material is not unknown. If the heat insulationthickness is 0, this indicates that a heating insulating material is notapplied.

[0029] The image data storage unit 20 stores image data obtained bytaking a picture of the plant by a picture-taking device such as acamera. Image data 22 and data of the position 21 and direction 22 inwhich a picture was taken, as shown in FIG. 4 for example, are stored inthe image data storage unit 20. The picture-taking position 21 shows thecoordinates of the camera used to take an image of the object. Thepicture-taking direction 22 shows the direction of the camera when theimage was taken. The image data is generated by using scanning data of acamera film, picture-taking data of a digital camera, or data on oneframe of a moving picture taken by a video camera, for example.

[0030] The parts library 30 stores information about the parts, such aspart models and image data, etc. Information about the kind 31, model32, picture-taking angle 33, image data 34, distance 35, and heatinsulating material 36, as shown in FIG. 5 for example, is stored in theparts library 30.

[0031] The model 32 shows particular identification names formed by acombination of a maker's name, a kind of part and a diameter, forexample. The picture-taking angle 33 denotes an angle of line of sightwhen the image data 34 was captured, and it is expressed by a pair of anazimuth angle and an elevation angle. With regard to the coordinateaxes, the flow direction is designated as the x-axis, the valve-stemdirection is designated as the z-axis, and the axis which isperpendicular to those axes and which extends on the right side asviewed from the flow direction is designated as the y-axis. The angle ofazimuth is an angle formed relative to the x-axis in the x-y plane, andthe angle of elevation is an angle formed relative to the x-axis in thex-z plane. The image data 34 stores image data on equipment whose imagewas taken at the specified picture-taking angle 33. The distance 35indicates a distance between the object equipment and the picture-takingangle. The setting of the distance 35 may be omitted. The heatinsulating material 34 indicates whether or not the image is of theequipment wound by a heat insulating material.

[0032] Data stored in the parts library 30 may be so arranged as to beupdated to latest data through communication lines, such as Internet. Inaddition, the parts library 30 may be located in a remote place andreferred to through communication lines.

[0033] The heat insulating material library 40 stores conditions forapplying a heat insulating material and the thickness of the materialused. Insulating material ID 41, service condition 42, floor height 43,upstream-side device 44, and heat insulation thickness 45, as shown inFIG. 6 for example, are stored in the heat insulating material library40.

[0034] The service condition 42 indicates the temperature condition ofthe internal fluid, the floor height 43 indicates the condition for theheight from the floor, the upstream-side device 44 indicates thecondition for the kind of part located upstream, respectively. The heatinsulation thickness 45 indicates the thickness of a heat insulatingmaterial applied when the conditions are met.

[0035] The heat insulating material library data may be edited by theuser or may be updated by entry of data through networks, such as theInternet.

[0036] The model correction unit 50 corrects model data based on imagedata, parts library data, and heat insulating material library data. Themodel correction unit 50 generates a synthesized image based oncorrected model data.

[0037] The user selects image data as the reference for comparison(hereafter referred to as reference image data) out of image data storedin the image data storage unit 20. The model correction unit 50determines object parts which are to be included in an image when animage is taken in the same position and direction as the picture-takingposition 21 and the picture-taking direction 22 of selected referenceimage data. In other words, the object parts are the same as those inthe reference image. With regard to the object parts, the modelcorrection unit 50 performs steps shown in FIG. 7 in the order fromupstream downwards.

[0038] By referring to the model data storage unit 10, the modelcorrection unit 50 decides whether or not the heat insulation thickness18 of the object part has been set (S101). If data has been set in theinsulation thickness 18 (including a case when 0 is set), the processproceeds to a step S107.

[0039] If there is set no data of the heat insulation thickness 18, themodel correction unit 50 decides whether or not the service conditionshave been set for the object part (S102). If the service conditions 17have been set, the model correction unit 50 refers to or searches intothe insulating material library 40, and decides the kind of a heatinsulating material and its thickness suitable for the part, andcorrects model data (S103, S104).

[0040] More specifically, the model correction unit 50 compares theservice conditions 17 for the part and the service condition 42 in theheat insulating material library (S103). If temperature has not been setin the service conditions 17, the temperature in the service conditions17 of a piping part one piece on the upstream side from the piping partconcerned may be substituted. The model correction unit 50 checks thefloor height 43 in the heat insulating material library to obtain afloor height from the reference points 14 of the part concerned. Inother words, if the reference point 1 of the reference points 14 of theobject piping part is (x1, y1, z1), data, included in model data andhaving CONC set as the kind of part, which is the first data tointersect a segment connecting (x1, y1, z1) and (x1, y1, z0), issearched. Here, CONC as the kind of part indicates that the floor of theplant is made of concrete. More specifically, the distance from thisintersection to the point (x1, y1, z1) is the distance from that part toits nearest floor, namely, the height of the floor. Note that thereference point 1 of a part concerned is designated as a reference usedas a base to measure its height though the part may sometimes beinclined. The model correction unit 50 identifies an upstream-sidedevice according to or from part ID 11, and collates it withupstream-side devices 44 in the heat insulating material library. Theupstream-side device is identified as a part which has an ID 11 thatincludes an element number smaller by one than that of the partconcerned. When an upstream-side device that meets the specifiedconditions, this is determined as a right upstream device, andaccordingly the kind and the thickness of a heat insulating material forthe part concerned are decided. If there are a plurality of heatinsulating materials that meet the conditions, a heat insulatingmaterial with a largest heat insulating material ID 41 is selected. Theinsulation thickness 45 in the heat insulating material library is setas the insulation thickness 18 of the part under discussion (S104). Ifthere is not any data on a heat insulating material that meets theconditions, the insulation thickness 18 is set to 0.

[0041] In the step S102, if no data has been set in the servicecondition 17, the model correction unit 50 carries out a process ofestimating the insulation thickness by using a reference image, andcorrects model data (S105, S106).

[0042] More specifically, model data is prepared which is applicable tothe object pipe, the pipe diameter of which is increased in 1-mmincrements (S105). Next, a synthesized image is generated based on thismodel data, and the synthesized image is collated with a referenceimage. One-half of an increase in pipe diameter when there is thehighest degree of similarity between the two images is set as theinsulation thickness (S106). The insulation thickness is 0 when asynthesized image based on model without any increase in pipe diameterhas the highest degree of similarity.

[0043] Here, based on process results up to the step S106, the modelcorrection unit 50 generates a heat-insulator-covered form model (S107).For example, a total of pipe diameter before correction added with twicethe thickness of a heat insulating material is substituted for a newpipe diameter.

[0044] Then, a synthesized image is prepared using model data correctedin the process up to the step S107 (Sl08˜S110). More specifically, forobject parts, such as valves, image data of which has been recorded inthe parts library 30, image data 34 is obtained from the parts library30 according to part models (S108). In the above step, either of imageshowing an object covered with a heat insulating material or imageshowing an object without any insulating material is obtained dependingon whether a heat insulator is used or not.

[0045] The image data 34 from the parts library 30 is corrected in termsof size or orientation in accordance with the reference image (S109).More specifically, the angle of line of sight is calculated based on thepicture-taking position 21 and the picture-taking direction 22 of thereference image, and the reference points 14 showing the position of theobject part. In other words, the angle of line of sight is expressed bythe horizontal and vertical components of an angle formed by a vectorconnecting the position of viewpoint (picture-taking position 21) andthe coordinates of the center of the object equipment, and thepicture-taking direction of image data, where the horizontal componentis an angle of azimuth and the vertical component is an angle ofelevation. By using this angle of line of sight, image data of the partcorresponding to this angle is obtained. In other words, the modelcorrection unit 50 searches the parts library 30 for an image whoseangle of line of sight coincides with or almost coincides with thepicture-taking angle 33 of the image of the part stored in the partslibrary 30. If there is not any such image, assuming symmetry of animage, an image for which the angle of azimuth is AZ±180° if the azimuthangle is designated by AZ or a left-right reversed image of an image forwhich the azimuth angle is 180-2*AZ is obtained as a result of search.

[0046] Thus, the obtained image is inserted into the synthesized image(S110). Projection transformation is carried out on model data so as tomatch the viewpoint position and the line-of-sight direction of imagedata to thereby obtain a two-dimensional image. At this time, thetwo-dimensional image as the search result is arranged at the centerposition in place of the part as the object. By those steps, based onmodel data, an image is synthesized (hereafter referred to as asynthesized image) reflecting changes in thickness of pipes and partsdepending on the presence or absence of heat insulating material andcontaining detailed features of the equipment.

[0047] The matching unit 60 collates synthesized image data generated bythe model correction unit 50 with reference image data selected by theimage data storage unit 20, and extracts differences. The matching unit60 reads reference image data and synthesized image data, and comparesthe density of the pixels of the two images. The density is compared bygray-scale representation of the colors of the pixels, and differencesin gray-scale value of the pixels are calculated. The matching unit 60conducts this calculation on all pixels, and determines the areas wherethe difference threshold value of 10 is exceeded among 5×5 or largeradjacent-pixel-areas, and outputs those areas as divergence areas. Theoutput method is to output an image formed of gray-scale values based onthe difference values, or an image formed by superposing an image havingthe gray-level values set in R values of RGB specification on the imagedata.

[0048] According to another embodiment of the present invention, it ispossible to provide three-dimensional data process service by usingimage matching service means which accepts image data or model data fromcustomers through networks, returns the customers collation resultsoutput from the above-mentioned matching means 60, and outputs customernames, process dates and times, and numbers of data.

[0049]FIG. 8 shows an example of a whole structure of a plantthree-dimensional image matching service system 8 as mentioned above.The plant three-dimensional image matching service system 8 comprises anetwork connection unit 70 and a matching service providing service 80in addition to the plant three-dimensional image matching system 1.

[0050] The network connection unit 70 conforms to the network datatransfer methods, such as e-mail, WWW, etc., connects to networks, suchas the Internet or an intranet, and outputs and receives data. The user,through this network connection unit 70, transmits a user name, imagedata of plant facilities, and plant model data. The image matchingservice providing process unit 80 receives image data and model datafrom the network connection unit 70, and stores the data in the modeldata storage unit 10 and the image data storage unit 20. The imagematching service providing process unit 80 activates the modelcorrection unit 50 and the matching unit 60, and obtains processresults.

[0051] The image matching service providing process unit 80 sends theuser image matching results through the network connection unit 70.Simultaneously, the image matching service providing process unit 80records the user name, the number of images processed, and process dateand time in the internal memory area of the service providing processunit 80. The recorded data, such as the user names, the numbers ofimages, and dates and times, can be output to the display unit at one'sdiscretion. Therefore, it is possible to check service executionrecords, and manage accounts of service fees.

[0052] The system structures mentioned in the fore-going embodiments maybe changed or added or omitted in any feasible combinations.

[0053] According to the present invention, differences between imagestrue to actuality of objects and model data can be extracted with highaccuracy.

[0054] It should be further understood by those skilled in the art thatthe foregoing description has been made on embodiments of the inventionand that various changes and modifications may be made in the inventionwithout departing from the spirit of the invention and the scope of theappended claims.

What is claimed is:
 1. An image matching device comprising: athree-dimensional model data memory unit for storing identificationinformation of each of a plurality of parts forming an aggregate andinformation of positions of said parts in a three-dimensional space; aparts data memory unit for storing data of said parts; an image datamemory unit for storing image data of an image taken of said aggregate,information of a position and a direction at which and in which saidimage was taken, the stored image data and the stored information of theposition and the direction being associated one another and stored;image synthesizing means for producing a synthesized image as viewedfrom a position and in a direction same as those of an image stored insaid image data memory unit by referring to said three-dimensional modeldata memory unit and said parts data memory unit; and means for matchingimage data of the produced synthesized image to image data of an imagestored in said image data memory unit.
 2. An image matching deviceaccording to claim 1, further comprising model correcting means forcorrecting three-dimensional model data stored in said three-dimensionalmodel data memory unit, wherein said image synthesizing means forgenerating a synthesized image by using three-dimensional model datacorrected by said model correcting means.
 3. An image matching deviceaccording to claim 2, wherein when a covering material is applied toeach of said parts, said three-dimensional model data memory unitfurther stores information of thickness of said covering material, andwherein said model correcting means corrects said three-dimensionalmodel data based on the information of the thickness of said coveringmaterial.
 4. An image matching device according to claim 2, furthercomprising a covering material information memory unit for storinginformation showing conditions for using a covering material for each ofsaid parts, wherein said three-dimensional model data memory unitfurther stores information of conditions for using each of said parts,and wherein said model correcting means corrects three-dimensional modeldata based on the information of said conditions for using said coveringmaterial by referring to said covering material information memory unit.5. A recording medium recorded with a program to be executed by acomputer and comprising: a process of obtaining three-dimensional modeldata including identification information of each of a plurality ofparts forming an aggregate and information of positions of said parts ina three-dimensional space, image data of an image taken of saidaggregate, image information including information of a position and adirection at which and in which said image was taken; a process ofproducing a synthesized image viewed from a position and in a directionsame as those of an image taken of said aggregate and based on saidthree-dimensional model data and said parts information; and a processof matching image data on said generated synthesized image to image dataon an image taken of said aggregate.
 6. A program for operating acomputer for execution of: a process of obtaining three-dimensionalmodel data including identification information of each of a pluralityof parts forming an aggregate and including information of positions ofsaid parts in a three-dimensional space, image data on an image taken ofsaid aggregate, image information including information of a positionand a direction in which said image was taken, and parts informationincluding image data on said parts; a process of producing a synthesizedimage viewed from a position and in a direction same as those of animage taken of said aggregate and based on said three-dimensional modeldata and said parts information; and a process of matching image data ofthe produced synthesized image to image data of an image taken of saidaggregate.